EasyNIC development

[insekt]

Hi everyone.
This issue is aimed to track the process of developing EasyNIC project.

The initial idea was introduced by @lukego . He also makes the spec https://github.com/lukego/easynic

Current HW basaline:

• FPGA - Lattice ECP5
• 10G PHY - Microsemi VSC8486
• 1 x 10G SFP+
• PCIe 2.0 x4

[insekt]

Feb 8.
I spent some time thinking of the NIC development. To eliminate the risks associated with ECP5 and PCIe soft IP core I suggest this plan.

1. Buy ECP5 PCIe devkit - http://www.latticesemi.com/Products/DevelopmentBoardsAndKits/ECP5PCIExpressDevKit
   I'll buy it.
2. Develop FPGA code according to your spec - https://github.com/lukego/easynic
   Suggested devkit doesn't have 10G link, so we emulate it in FPGA - for example, we can make a simple loop in FPGA between Tx and Rx right after MAC IP core.
   My friend has agreed to help with it.
3. Develop the driver according to your spec.
   @lukego agreed to help with it.
4. Make complex performance tests
   I will setup server with devkit and provide remote access to it so we can do tests.

[insekt]

Feb 12.
Small update on EasyNIC.

1. ECP5 devkit with PCIe x4 is on the way to me
   http://www.latticesemi.com/Products/DevelopmentBoardsAndKits/ECP5PCIExpressDevKit
2. I got developer docs for Microsemi 10G PHY VSC8486, schematic and pcb ref designs, etc. It seems pretty good. Need some time to deep dive into details.

[insekt]

Reading through the user guide for ECP5 IP core, the max performance of the core is 8 Gbit/s, either in x2 or x4 case. In x2 transceivers can work at 5 Gbit/s as result we get 1 GB/s or 8 Gbit/s. In x4 transceivers can work only at 2.5 Gbit/s as the result we get the same 1 GB/s or 8 Gbit/s.

x2
5 GT/s x 8/10 bit/Т = 4 x 10^9 bit/s = 4 Gbit/s = 0.500 GB/s
0.500 GB/s x 2 lanes = 1 GB/s = 8 Gbit/s

x4
2.5 GT/s x 8/10 bit/Т = 2 x 10^9 bit/s = 2 Gbit/s = 0.250 GB/s
0.250 GB/s x 4 lanes = 1 GB/s = 8 Gbit/s

@lukego Does it suitable for 10G NIC?

But for all that if we agree to use x2, we may try to use PCIe port bifurcation, split x4 to 2 by x2 and put 2 x 10G PHY on one board.

Two datapaths in one FPGA:

                              ECP5-5G
           ----------------------------------------------
10G PHY - | 10G MAC - Custom IP core - PCIe IP core - x2 | - \
          |                                              |    > PCIe x4
10G PHY - | 10G MAC - Custom IP core - PCIe IP core - x2 | - /
           ----------------------------------------------

It is the only idea for now.

[lukego]

I wonder if a better PCIe core is available that can do 16G? I'm asking the Twitterverse.

PCIe bifurcation should be something that we can live with if we have to but I think (?) that significantly restricts the set of compatible motherboards/slots.

P.S. Does using the Lattice PCIe core also lock us into their proprietary toolchain instead of Yosys/nextpnr? How are the licensing costs for people deploying EasyNIC?

[lukego]

(also: sounds plausible to ship an EasyNIC with 8Gbps PCIe bandwidth if this can later be upgrade to 16Gbps with a firmware update (new PCIe endpoint core). This is a bit like the way Mellanox bring hardware to the market quickly, it's hair-raising to see all the stuff that is fixed in their early firmware updates!)

[insekt]

I wonder if a better PCIe core is available that can do 16G? I'm asking the Twitterverse.

I didn't find anything. Also if we will find anything I doubt that it would perform better than IP core from chip manufacturer.

PCIe bifurcation should be something that we can live with if we have to but I think (?) that significantly restricts the set of compatible motherboards/slots.

Yep. Agree with that.

P.S. Does using the Lattice PCIe core also lock us into their proprietary toolchain instead of Yosys/nextpnr?

I don't know at all. Me or somebody whom I know have never tried to use such toolchain. It would be great to get into this conversation people from these projects.

How are the licensing costs for people deploying EasyNIC?

PCIe IP core is licensed per project. If we buy it for this project I think everybody can use it. But I'm not 100% sure.
This varies from $1000 to $3000, looked at Digikey, Mouser, etc. We may also try to request a free license, I think for open project Lattice may give it for free to promote ECP5 FPGAs.

(also: sounds plausible to ship an EasyNIC with 8Gbps PCIe bandwidth if this can later be upgrade to 16Gbps with a firmware update (new PCIe endpoint core). This is a bit like the way Mellanox bring hardware to the market quickly, it's hair-raising to see all the stuff that is fixed in their early firmware updates!)

To make it possible it is necessary to write PCIe IP core from scratch or modify some opensource (Lattice IP core is provided as "binary"), it's very hard work. And I think it's not achievable in principle. If it was possible then Lattice would already implement it.

[insekt]

Having in mind restrictions and risks with ECP5 and 10G PHY, I would like to suggest an alternative way for EasyNIC.
I looked for the available boards based on Xilinx and Intel which is suitable to the initial request - make NIC with 1 or better 2 10G ports and I found one on Aliexpress:
https://www.aliexpress.com/item/xilinx-board-xilinx-fpga-board-xilixn-fpga-development-board-pcie-board-Kintex-7-XC7K420T-XC7K325T-xilinx/32907109444.html
It based on Kintex 7 K420T FPGA.
It has 4 ports of 10G.
It has built-in 10G transceivers.
It has hard IP core PCIe 2.0 x8 (32 Gbit/s)
It provides full access to FPGA.
It costs around $170.
The manufacturer has pretty good feedback from people from all around the world.

Maybe we will consider this board as a starting point? Or at least we can do it parallel. While we considering ECP5, we can start writing code for this board. The main FPGA code - 10G MAC, the custom IP core (to glue 10G MAC and PCIe) - should remain the same and the driver too, I think.

[mithro]

Have you considered using LiteEth instead of EasyNic? It would good to see a comparison of the two cores.

LitePCIe is also in the process of being ported to the ECP5.

[insekt]

Hi @mithro .
About LiteEth, no I didn't. I definitely will look at it.
Do you think it's possible to get 16 Gbps on ECP5-5G with LitePCIe?

[mithro]

@insekt The limitation is not going to be LitePCIe, it'll be the transceivers and other hard blocks.

[insekt]

I almost get ECP5 devkit with PCIe x4. It has LFE5UM-85 chip with 3 Gbps transceivers. I can swap the chip to LFE5UM5G-85 with 5 Gbps transceivers (they pin-to-pin compatible). Need to check devkit schematic to be 100% sure. So it's possible to prototype.

[lukego]

@mithro LiteEth/LitePCIe looks really interesting, great if there is something there that we can reuse!

EasyNIC is all about making software developers happy by creating a simple and beautiful interface for device drivers. So we need to have control over the configurations registers and DMA model exposed to the host -- don't want to reuse that part off the shelf. But that's only a small part of the puzzle.

@insekt It's not a hard requirement but I would love for the finished FPGA NIC to be supported by Yosys/nextpnr. This way software people would be tempted to build their own firmwares and potentially make adaptations for their own applications by forking the firmware ("OpenWRT" style.) The reason I am attracted to ECP5 over Xilinx/Intel is the good open source toolchain support.

Just speaking for myself, I have once in my life setup all the Xilinx proprietary developer tools, and I don't think I will ever bother to do that again. It's a major culture clash for software people who are used to gcc/clang/etc and expect to just do git clone && make.

(also: the genesis of the EasyNIC project is activism. NIC vendors are making everything complex and proprietary and pushing people into their massive software ecosystems like DPDK. This sucks and we need a "NIC for hackers" to push back on that. So the project is philosophically aligned with Yosys/nextpnr/etc who are pushing in the hardware/FPGA space.)

[insekt]

@lukego I share you point.

I looked at YosysHQ/nextpnr, they hope to get support of Xilinx 7 series with help of https://github.com/SymbiFlow/prjxray .

As I said before the major problem with ECP5 based EasyNIC is XGMII 10G PHY. It is outdated technology and there is only one chip from one vendor on the market right now. And it is 10 years old. I'm waiting for reply from Microsemi (BTW, now part of Mircochip) about EoL/EoS. I don't think there will be any other vendor who will be release a new chip with XGMII interface.

Please don't get me wrong but I'm trying to be realistic and think what will be if some day VSC8486 disappears from stocks.

[lukego]

@insekt Great that you are getting information about the PHY. If the lifetime is very short that would be a problem.

On the other hand 10G ethernet and PCIe 2.0 are also basically "outdated technology." Sooner or later we will need to update to 25G ethernet and PCIe 3.0. Maybe over the next few years? So it could make sense to build the 10G NIC using simple "trailing edge" technology and count on rethinking choice of FPGA, PHY, etc in the future.

Of course if the VSC8486 is already unobtainium next year then that would not be much fun...

[mithro]

Quick question - Do you have a working PCIe 1.0 + GigE (or maybe quad-GigE) version? Would it make sense to start somewhere there and then move towards the 10G / 25G standards once the basics work?

[lukego]

@mithro Goal here is to make a NIC that Snabb users (for example) can deploy applications with. These are typically people working in ISPs or other network operators. In this context 10G and PCIe 2.0 is acceptable at the moment but 1G would be too awkward for most people.

The attraction of starting directly with 10G, to me, is that we de-risk important aspects of the project early on. For example, if there is not really a working 10G FPGA+PHY combo that will be available for the next few years then we might be heading for a dead-end and need to rethink the whole project (e.g. switch from FPGA to SoC, etc.)

[insekt]

My current thoughts on the topic.

1. I found a way how to deal with the 10G PHY problem. We can use 10G PHY with XAUI interface but additional ECP5 will be necessary.
   10G PHY (Marvell 88X3310) - XAUI - ECP5 - XGMII - ECP5-5G - PCIe x4

But we still have problems:

• IP soft core PCIe x4 for ECP5-5G that is capable to handle 10Gpbs
• IP soft core XAUI<->XGMII is required (a quick search didn't show any open source version)
  Lattice has one https://www.latticesemi.com/en/Products/DesignSoftwareAndIP/IntellectualProperty/IPCore/IPCores01/XAUI10GbEthernetAttachment
  Costs about $1000.

Additional ECP5 will also add some cost.

For now, IP soft core PCIe x4 for ECP5-5G seems to be impractical to be developed. At least from my point of view. If someone can invest his time to this work I can help him with HW. I don't know anybody who can help with developing of the IP core.

2. Suggested Xilinx 7 PCIe card looks nice. Now looking through the ref. projects from the board's developer.
   http://www.mediafire.com/file/x44lzh3woq5bdln/XC7K420T_core_sfp_pcie.rar/file

Yeap, there is not open source or free IDE for Xilinx Kintex 7. ICE WebPACK doesn't support it. But I think for the current stage of the EasyNIC project it's not so critical.
The main FPGA code will be open source. I'm sure it is more valuable.
For the snabb user this board will be like Intel 82599, just the NIC with 2x10G interfaces but the driver is open and low-level firmware is open. Maybe also it will force adding support of Xilinx 7 Series in YosysHQ/nextpnr.

Working with Xilinx also opens up the doors to 25G and 100G cards.

rethink the whole project (e.g. switch from FPGA to SoC, etc.)

@lukego How do you imagine that? What kind of SoC? NPU or what?

[lukego]

10G PHY (Marvell 88X3310) - XAUI - ECP5 - XGMII - ECP5-5G - PCIe x4

Awesome!

So the idea is that one ECP5 talks high-speed SERDES to the 10G PHY, the other ECP5 talks high-speed SERDES to PCIe, and then between the ECP5s we have lower-speed (more parallel) connection?

IP soft core XAUI<->XGMII is required

Can go fishing on Twitter perhaps...

Guessing that really any 10G ECP5-to-ECP5 I/O interface would be okay provided it doesn't need SERDES? (Since we control the code on both sides we are not locked into a standard, even though the standard might represent the best solution.)

IP soft core PCIe x4 for ECP5-5G that is capable to handle 10Gpbs

Suggest we start with PCIe 1.0 x4 (8Gbps) and plan to upgrade to PCIe 2.0 x4 (16Gbps) in a future firmware update.

Have you looked at LiteEth and LitePCIe btw? I'm thinking it would be wonderful if we could use open-source IP cores instead of vendor ones (especially since people seem to be releasing such cores lately and presumably looking for users of them?)

Additional ECP5 will also add some cost.

ECP5s cost around $5 for low-end up to $50 for high-end right? This seems quite reasonable to me.

Yeap, there is not open source or free IDE for Xilinx Kintex 7. ICE WebPACK doesn't support it. But I think for the current stage of the EasyNIC project it's not so critical.

This seems like the toughest decision for the project to me. The ECP5 version would have less I/O bandwidth but end-users might realistically hack the firmware to add features and fix bugs ("OpenWRT style"). The Xilinx version would provide the simple EasyNIC driver interface but otherwise be more like a traditional/opaque NIC (I don't think software people will think Xilinx tools are fun.)

I have a strong preference for the hackable ECP5 variant myself but I might be unusual here since I'm mostly thinking about lab use rather than deployment. @alexandergall as somebody who deploys his own applications in a production environment do you have thoughts on which EasyNIC would be more interesting?

Working with Xilinx also opens up the doors to 25G and 100G cards.

This is definitely important in the future. Question is timing though. Just now the Xilinx FPGAs don't have mature open source toolchain support and so to me it seems tempting to wait until they do before adopting.

How do you imagine that? What kind of SoC? NPU or what?

No, I have no specific ideas in this area, I only mean that in past discussions about EasyNIC it did not sound like FPGA was a viable solution and so "something else" would be needed instead. That was until @daveshah1 pointed out that ECP5-5G has enough SERDES to do 10G.

[insekt]

Read the Marvell 88X3310 datasheet carefully.
With XAUI on the host side (towards FPGA) - it supports only 10GBASE-T (copper, RJ-45) on the line side.

[insekt]

Guessing that really any 10G ECP5-to-ECP5 I/O interface would be okay provided it doesn't need SERDES?

Yes, but it's better not to reinvent the wheel =)

ECP5s cost around $5 for low-end up to $50 for high-end right?

I think $5 is the cost of ECP5 without transceivers at all. ECP5 with 3G transceivers $15-20, I guess.

[insekt]

It seems that only Marvell 88X2222 supports XAUI for host and XFI for line (SFP+). But this PHY provides min. 2 ports 10G.

[lukego]

Yes, but it's better not to reinvent the wheel =)

Right. I only mean that if we are looking for an existing open-source core then we might find something that works even if it was not designed for ethernet.

[lukego]

But this PHY provides min. 2 ports 10G.

If we were "stuck" with a second SFP+ then we could connect that up to the FPGA for future use.

Myricom used to offer NICs with 1xMAC and 2xPHY that would automatically use the second port as a backup when link was lost on the first port. This is quite nice if you want to connect a server to redundant switches and handle failover transparently in hardware. This is not something we'd need to implement but it would be fun as a potential feature to be added by a user.

(Aside: I miss Myricom NICs. They did such a great job of their technology. I hope to see more cool stuff from those people one day.)

[insekt]

I have bad news =(
I won't get ECP5 PCIe devkit - http://www.latticesemi.com/Products/DevelopmentBoardsAndKits/ECP5PCIExpressDevKit
Here is the story. I ordered devkit from a local reseller, who in turn bought the devkit from one of the global stock. When my local reseller got the parcel he saw that the original package was seriously damaged and there were signs of water on the package and corrosion on the board (it seems that that the package with the board spent some time in a very wet place). What is the saddest thing is that it was the last devkit in the stock. I have decided to not accept such parcel and requested for refund.
I spent quite a lot of time looking for other ways to buy it but not yet achieved success.

So now, it is a turning point, the only way to move on with ECP5 is to design and manufacture custom board.

[lukego]

Ouch!

@daveshah1 is it really true that PCIe ECP5 dev boards are not available anymore??

[insekt]

@lukego
ECP5 dev board are still available but this board is the only one with PCIe x4. All other boards comes with x1.

[lukego]

@insekt And from your past research one potential alternative is a Kintex7 board with PCIe x8 and 4xSFP+ from Aliexpress for $166? https://www.aliexpress.com/item/xilinx-board-xilinx-fpga-board-xilixn-fpga-development-board-pcie-board-Kintex-7-XC7K420T-XC7K325T-xilinx/32907109444.html

[insekt]

@lukego yes.
We still have an option to design and develop ECP5 based NIC but there are a lot of risks which we unable to research.

Speaking of Microsemi 10G PHY VSC8486, I didn't get any answer for Microsemi but several official resellers answered that there is no plan for EOS/EOL.

[lukego]

@insekt What do you think about using an ECP5 board with PCIe x1 for prototyping and derisking? Could this be sufficient for everything except the full-speed PCIe core? (Guessing we could "amplify" internally to drive the MAC/PHY at full speed e.g. by quadruplicating each packet on the transmit path?)

[insekt]

@lukego I do not have doubt that ECP5 will work as specified in Lattice docs:
2.5G - x1, x4 modes
5G - x2 mode
If the vendor who develops FPGA and PCIe IP core for this FPGA doesn't claim support of 5G x4 mode then for me, it seems suspicious. The thing that is obvious to be implemented, is missed.

I planned to verify exactly this point on the devboard with PCIe x4 by swapping ECP5 to ECP5-5G (and some modification in power system). What I'm thinking now is to make a simple board - just ECP5-5G, PCIe x4, Flash memory, JTAG.

I also found this board https://www.sanitaseg.com/project/singrab-board/
that also seems to be a good candidate for experiments. I sent an email with request twice (in Jan and Feb) but didn't get the answer.

[insekt]

Finally, I got reply about this board https://www.sanitaseg.com/project/singrab-board/ =)
The board is available, so I can buy it and make PCIe x4 performance testing to de-risk this part of ECP5.
But before I'd like to make sure that this work is still relevant. I meant does it make sense to make 1-port 10 Gpbs FPGA-based (Lattice ECP5) NIC?
@lukego please tell, what is your current view on that topic?

[lukego]

Cool!

My feeling is that the situation is the same as it was six months ago. Those few of us who wanted it then still want it now :-). I'm planning to finally make a deep dive into HDL coding over the coming months and I see contributing to this NIC as a good chance to get started on that. I'm currently reading the Chisel book :)

[lukego]

(I suspect the major development that would affect the direction of this project is for Xilinx and/or Altera to get sufficiently solid open source toolchain support to make their hardware more attractive. I think that's a couple of years off yet but we'll see.)

[eugeneia]

Hey that board seems really cool. I’m really close to Milan (where Sanitas EG appears to be) at the moment (just an hour with the train away.) Maybe I’ll walk into their office some day and try to get one too. :-)

[lukego]

Can you really buy that Sanitas board? How and what does it cost?

[janrinze]

@insekt : did you get the board from Sanitas EG?
Very curious how it goes and if it was similar priced to the one from Lattice.